Repository navigation
h2: a loop is taken as stuck after half a WriteTimeout and at least 2 s (the wedge fallback cut healthy downloads on a loaded host) - #1202
Merged
Conversation
…mp; the many-downloads cell asserts the condition (red on main) Refs #1170. Coverage root shard c, run 41246492268: TestH2GoTransportManyParallelDownloadsAreNotCut1170/adaptive had 7 of 8 downloads reset (loop-blocked 4, rate 3). The loop-stuck fallback judges a loop stuck from the wall-clock age of its current batch (max(WriteTimeout/2, 250 ms)); a healthy loop of an instrumented or overloaded host has batches of 0.4 to 1.9 s (one h2 connection's write-queue drain), measured at 7 of 72 cells with -race, -covermode=atomic and two busy loops on two CPUs. Unit tests, on mono.Manual (exact, no sleep): - loopStuckFor has a 2 s floor (1 s -> 2 s, 5 s -> 2.5 s, 60 s -> 30 s unchanged); - a batch of 0.8, 1.5 and 1.9 s at WriteTimeout 1 s is not a wedge, 2 s is; at 5 s and at the 60 s default the edge is 2.5 s and 30 s; a waiter that has not waited a WriteTimeout is never cut by it; - a loop that blocks mid-wait is cut within [threshold, threshold + WriteTimeout] (the positive control, with the bound stated); - a loop that ends without returning (runtime.Goexit, celeris#848) leaves its LoopWatch stamp set: every waiter of the engine would be cut as loop-blocked for ever (epoll and io_uring, in the 848 exit tests). Cells: the many-downloads cell runs the engines one at a time and asserts that no wait was reset as loop-blocked before it asserts the downloads; the wedge cell asserts the bound as a formula; the threshold pair blocks a loop for 2.5 s (over) and 0.1 s (under). Red on this commit's parent code: the floor, the healthy-batch cases, the stale stamp. Green: the bound, the 60 s edge, the cells.
… s; a loop that dies leaves no stamp Refs #1170. The wedge fallback (LoopWatch) cuts every pool-handler wait that has lasted a WriteTimeout while any event loop has been in one batch for max(WriteTimeout/2, 250 ms). A healthy loop of an instrumented or overloaded host has batches of 0.4 to 1.9 s (one h2 connection's write-queue drain), so the cell of eight parallel downloads at WriteTimeout 1 s lost streams as loop-blocked (Coverage root shard c, run 41246492268). The floor is now 2 s (minLoopStuck): unchanged for a WriteTimeout of 4 s or more (the 60 s default keeps its 30 s), and a wedge is cut at most max(WriteTimeout/2, 2 s) + WriteTimeout after the loop blocks (3 s at WriteTimeout 1 s, was 1.5 WriteTimeouts). No hot-path cost. A loop or worker that ends without returning (runtime.Goexit in a sync handler, celeris#848) now clears its LoopWatch stamp on the way out: it was left set, and every waiting handler of the engine was cut as loop-blocked from then on. Docs: Config.WriteTimeout, the OutboundBudget comment and H2Config.LoopWatch state the threshold and bound.
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
… exit clears the stamp first Refs #1170. Review r1 of #1202: - MINOR-1: BlockedFor(now+1h, 1) is true when ANY slot of the engine is stamped, so a live loop caught in a batch at that instant read as the dead loop's stamp (a flake shape on a starved runner). The tests find the loop or worker the /boom ran on (boomID848) and check its own slot (LoopSlot.Stamped). - NIT-1: the Leave ran after reportLoopExit/reportWorkerExit, whose Logger can block while the stamp ages and the engine's waiters are cut as loop-blocked. It is now the first statement of the deferred function.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1170
What failed
Coverage root shard cof #1193 (run 41246492268, attempt 1; the rerun passed):TestH2GoTransportManyParallelDownloadsAreNotCut1170/adaptive, 7 of the 8 parallel 8 MiB downloads reset with INTERNAL_ERROR at 4 MiB (server log: resets by reasonloop-blocked:4 rate:3, 1 episode of a stuck loop). It had failed three times on the l19s3 branch before.Cause
The wedge fallback of #1180 (
LoopWatch) cuts every pool-handler wait that has lasted a WriteTimeout while any event loop has been in one batch formax(WriteTimeout/2, 250 ms), judged by a single sample of the batch's wall-clock age at the waiter's own 1 s timer. A healthy loop of an instrumented or overloaded host has batches of 0.4 to 1.9 s, almost all of it one h2 connection's write-queue drain (drainH2Writes; the 8 downloads share one connection). So a loop that is slow, not blocked, switched the rate rule off and the 8 waiters, whose timers fire together, were cut together.Reproduced in docker arm64 (
--cpus 2,GOMAXPROCS=2,-race -covermode=atomic -coverpkg=..., the CI shard's flags): 0/48 subtests with no extra load, 0/48 with one busy loop, 7/72 with two busy loops (every oneloop-blockedfirst;stuck_for0.64 to 1.8 s against the 0.5 s threshold; the CI signatureloop-blockedthenratereproduced). Without-raceor coverage the longest batch under the same load was 324 ms.Fix
minLoopStuck250 ms -> 2 s: a loop is taken as stuck aftermax(WriteTimeout/2, 2 s). Unchanged for a WriteTimeout of 4 s or more (the 60 s default keeps its 30 s). No hot-path cost, portable. The price is the bound of a real wedge at a small WriteTimeout:max(WriteTimeout/2, 2 s) + WriteTimeoutafter the loop blocks (3 s at WriteTimeout 1 s, was 1.5 WriteTimeouts = 1.5 s; measured by the wedge cell 1.3 s before, 2.3 s expected after).runtime.Goexitin a sync handler, epoll, io_uring: runtime.Goexit in a synchronous handler ends the loop or worker goroutine it runs on, and every connection that loop owns goes unserved #848) now clears itsLoopWatchstamp on the way out. It was left set, and every waiting pool handler of that engine was cut asloop-blockedfor the rest of the process.Config.WriteTimeout, theOutboundBudgetcomment,H2Config.LoopWatch.loop-blockedbefore it asserts the downloads; the wedge cell asserts its bound as the formula; the threshold pair blocks the loop for 2.5 s (over) and 0.1 s (under).Considered and rejected (design review): a thread-CPU/run-queue verdict (
schedstat) is unsound under Go (a runnable loop goroutine without a P has a sleeping thread, the same signature as the wedge: two healthy batches of 801 and 835 ms read ~0), and a progress counter has healthy CPU-busy batches of up to 4.1 s without a bump (one frame's growslice/GC assist).Failing first
Commit 1 (tests only) is red on the parent code:
TestLoopStuckThresholdHasAFloor1170,TestHealthyLongBatchIsNotAWedge1170(0.8, 1.5, 1.9 s at WriteTimeout 1 s),TestWaiterLoopBlockedFallback1170/just_under,TestLoopWatch1170, and the Goexit stale-stamp assertions in the epoll and io_uring 848 tests. Onmono.Manual, so the verdicts are exact. Green before and after:TestBlockedLoopIsCutWithinTheBound1170(the wedge is still cut, within[threshold, threshold + WriteTimeout], at WriteTimeout 1 s, 5 s and 60 s) and the 60 s edge.Verification
Docker arm64 (kernel 7.0.12-linuxkit),
--cpus 2,GOMAXPROCS=2,--security-opt seccomp=unconfined --ulimit memlock=-1, golang:1.27.3, one container at a time; the test binary is built as the coverage job builds it (go test -c -race -covermode=atomic -coverpkg=github.com/goceleris/celeris ./test/server) and run as-test.run '^TestH2GoTransportManyParallelDownloadsAreNotCut1170$' -test.count=N; "busy loops" arewhile :; do :; donein the same container (noisy neighbours). One iteration = the epoll and adaptive subtests (a failure is counted per subtest). Logs and the driver scripts: the FLAKES9 evidence directory.The cell, main against the fix (the cell unchanged, both subtests in parallel, unless stated):
stuck_for0.64 to 1.8 s; CI shape (loop-blockedthenrate) seen onceloop-blocked:2,loop-blocked:5 rate:1,loop-blocked:2; 1 episode eachThe baseline did not reproduce at two busy loops this session (0/72 on main, against 7/72 in the earlier session, which was run with a build instrumented per batch; the host load differs from one session to the next). At three busy loops it did (3/24 on main, all
loop-blocked), and the fix is 0/24 at the same load with the cell unchanged, so the fix is verified independently of the cell change. The deterministic evidence is the red/green unit tests below, not these rates. (3 failures in 24 against 0 in 24 is p about 0.1 by itself.) The Opus review ran an extra interleaved A/B at three busy loops with the no-loop-blocked assertion on both sides: main 3/32 failed (one with the exact CI signatureloop-blocked:4 rate:3), this head 0/32; pooled with the 3/24 against 0/24 above: main 6/56 against 0/56, p = 0.014. The 2-busy-loop rows of the fix show only that nothing regressed; the evidence that the fix matters is the 3-busy-loop rows, the pooled A/B and the red/green unit tests.Wedge and threshold cells (this head, 2 busy loops, 3 iterations of both tests):
TestH2PoolWaiterHoldingALockDoesNotWedgeTheServer1170: the locker is answered in 2.30 to 2.32 s on epoll, io_uring and adaptive, all 24 trials per engine (the assertion is the formulamax(WT/2, 2 s) + WT + 1 s= 4 s; main measured 1.30 s). Wedge bound at WriteTimeout 1 s: 3 s.TestH2PoolWaiterLoopBlockedThreshold1170:under(0.1 s blocks): 0 cut, 0 loop-blocked resets, epoll and adaptive;over(2.5 s blocks): 3 of 4 cut,loop-blockedand an episode told, both engines. The test takes 59 s (was 54.6 s in the CI log of iouring: the header deadline of a silent connection is a kernel timer armed at the accept; document the bound (Refs #1189) #1193).TestSyncHandlerGoexitEndsOneLoopLoudly848,...OneWorkerLoudly848, which check the dead loop's own slot;-race -count=20in the container: 20/20 on both engines): red on the tests-only commit ("the loop that ended without returning left its LoopWatch stamp set"), green on the fix, both engines.-race -count=20clean, clean withCELERIS_MONO_SKEW=3600sand-3600s(-tags celeris_monoskew); the whole stream package-racepasses.1170cells of./test/serveronce under-race -covermode=atomicin the container: pass, 494.7 s (the many-downloads cell is 67 s now, serialised, was 34 s: shard c has 660 s of timeout for about 360 s of work, so +34 s and +4 s stay inside the headroom).golangci-lintclean on the changed packages (two errcheck hits in unchanged files of other packages come from a newer local lint version than CI's).Release note
HTTP/2 pool-handler waits: the fallback that bounds a waiting handler by WriteTimeout while an event loop is blocked (#1170) now needs the loop to be in one batch for half a WriteTimeout and at least 2 s (was 250 ms). A loop that is merely slow (an overloaded or instrumented host) no longer switches the minimum-rate rule off; the waits of a handler that holds a lock a blocked loop waits for are still cut, at most
max(WriteTimeout/2, 2 s) + WriteTimeoutafter the loop blocks (1.5 WriteTimeouts for a WriteTimeout of 4 s or more, 3 s at WriteTimeout 1 s); the 60 s default is unchanged. A loop that ends abnormally (runtime.Goexitin a handler) no longer leaves the fallback switched on for the rest of the process (the waits of that engine were all cut as loop-blocked).Follow-up (not in this PR)
drainH2Writesappends up to 8 x 4 MiB into one connection'swriteBufin one pass (a 32 to 64 MiB growslice, its copy under the race detector, the GC assist it pays, thenflushWrites' compaction copy), which is where the long batches come from. A bounded pass (stop pulling from the shard queue past a few MiB unsent, resume on EPOLLOUT / the next pass) would shorten the batches and the per-connection memory by an order of magnitude, but touches write-path accounting (pendingBytes/writeCaprefusal, the drain clock) and io_uring's queue-to-SQE path. To be filed as a performance issue.